Display controlling apparatus, display controlling method, and storage medium

ABSTRACT

There is provided a display controlling apparatus which comprises: an obtaining unit configured to obtain virtual camera path information related to a movement path of a virtual viewpoint related to a virtual viewpoint video image generated based on a plurality of shot images obtained by shooting a shooting target area with a plurality of cameras; a generating unit configured to generate a virtual camera path image representing the plurality of movement paths including first and second movement paths of the virtual viewpoint, based on the virtual camera path information obtained by the obtaining unit; and a display controlling unit configured to display the virtual camera path image generated by the generating unit, on a display screen.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a technique of displaying, on atwo-dimensional plane, trajectory data (camera path) of a virtual camerain a three-dimensional space.

Description of the Related Art

In computer graphics (CG), a three-dimensional object modeled inthree-dimensional space is rendered as a two-dimensional video imageviewed from an arbitrary viewpoint. A camera virtually disposed in thethree-dimensional space to express position and visual field range(angle of view) of an arbitrary viewpoint is called a virtual camera. Anoperator of the virtual camera manipulates the position and visual fieldrange of the virtual camera in the three-dimensional space according toprogress of a scene. Trajectories (or loci) of the position and visualfield range of the virtual camera in the three-dimensional space aregenerally called virtual camera paths (or simply camera paths). Besides,a two-dimensional moving image generated based on the camera path iscalled a free viewpoint video image. If there are three-dimensionalobject data of a scene and camera path data, it is possible to reproducethe trajectories of the position and visual field range of the virtualcamera to play back the free viewpoint video image.

In recent years, a technique of generating a three-dimensional objectfrom video images shot by a large number of cameras (actual camerasinstead of virtual cameras) and playing back a free viewpoint videoimage has been developed, and an expectation for using the played-backfree viewpoint video image for various needs has increased. Inparticular, in sports broadcasting, there are great needs for replayingone scene from various viewpoints (even from a viewpoint where there isno actual camera), so that a large number of camera paths are oftencreated from one scene. For example, in live broadcast of sports, alarge number of camera paths respectively viewed from differentviewpoints are generated in a score scene and a fine play scene, andfree viewpoint video images are replayed one after another. Moreover,the score scene and the fine play scene are replayed many times duringthe game and after the game, so that there is a case where the camerapath is reused.

Here, since time is limited in the live broadcasting, it is required toquickly and surely find a desired camera path from among a large numberof created camera paths and to play back the video image using the founddesired camera path. On the other hand, in a highlight program to bebroadcasted at a later date, there is a possibility that more camerapaths are created from one play and various free viewpoint video imagesare played back. Also, in the highlight program, since there is timefrom a game to broadcasting, more camera paths are created by trial anderror in addition to the camera paths actually used for the broadcastingin order to pursue better camera paths. That is, in the highlightprogram, since a great number of camera paths are created, it isrequired to quickly find the camera path to be reedited and/or thecamera path for the free viewpoint video image playback from among thecreated camera paths. Regarding such a point, Japanese PatentApplication Laid-Open No. 2014-164685 discloses that free viewpointvideo images respectively played back from a plurality of camera pathsare simultaneously displayed on a plurality of screens.

According to Japanese Patent Application Laid-Open No. 2014-164685, auser (operator) looks at the free viewpoint video image to be playedback and decides whether or not a desired camera path has been used.However, in a case there are a plurality of similar camera paths, itoften takes much time until a scene by which the user can decide whetheror not the desired camera path has been used in the free viewpoint videoimage appears. Besides, in a case where the user looks at a large numberof free viewpoint video images at the same time, there is a fear thatthe user misses a scene by which he/she can decide whether or not thedesired camera path has been used. In this case, if the user misses therelevant scene, he/she has to perform the playback again. Besides, in acase where there are a large number of camera paths and it is impossibleto display the free viewpoint video images at once, it is necessary toplay back the free viewpoint video images in plural times. In this case,it takes time and it is difficult to perform comparison of these videoimages. That is, in the related background art, it is difficult toquickly find a desired camera path from among a large number of camerapaths.

SUMMARY OF THE INVENTION

The present invention has been completed in consideration of such aproblem as described above, and an object thereof is to provide adisplay controlling apparatus which is characterized by comprising: anobtaining unit configured to obtain virtual camera path informationrelated to a movement path of a virtual viewpoint related to a virtualviewpoint video image generated based on a plurality of shot imagesobtained by shooting a shooting target area with a plurality of cameras;a generating unit configured to generate a virtual camera path imagerepresenting the plurality of movement paths including first and secondmovement paths of the virtual viewpoint, based on the virtual camerapath information obtained by the obtaining unit; and a displaycontrolling unit configured to display the virtual camera path imagegenerated by the generating unit, on a display screen.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a camera path displaying system accordingto an embodiment of the present invention.

FIG. 2 is a flowchart of a camera path list displaying process accordingto a first embodiment.

FIGS. 3A and 3B are diagrams for describing data formats of camerapaths.

FIGS. 4A and 4B are diagrams for describing a first screen example ofcamera path list display.

FIG. 5 is a diagram for describing a modified example of the firstscreen example of the camera path list display.

FIG. 6 is a diagram for describing a second screen example of the camerapath list display.

FIGS. 7A and 7B are diagrams for describing a third screen example ofthe camera path list display.

FIG. 8 is a diagram for describing a fourth screen example of the camerapath list display.

FIG. 9 is a diagram for describing a modified example of the fourthscreen example of the camera path list display.

FIG. 10 is a diagram for describing another modified example of thefourth screen example of the camera path list display.

FIG. 11 is a flowchart of the camera path list displaying processaccording to a second embodiment.

FIG. 12 is a diagram for describing a screen example of the camera pathlist display according to the second embodiment.

DESCRIPTION OF THE EMBODIMENTS

Preferred embodiments of the present invention will now be described indetail in accordance with the accompanying drawings. Incidentally, it isto be noted that the embodiments described below show an example in acase where the present invention is concretely practiced, and thepresent invention is not limited to the following embodiments.

In the present embodiment, there will be described an example in whichan information processing apparatus of the present invention is appliedto a camera path displaying system in which trajectories of positionsand visual field ranges (angles of view) of a plurality of cameras in athree-dimensional space are displayed as camera paths on atwo-dimensional plane.

First Embodiment

FIG. 1 is a block diagram of a camera path displaying system includingan information processing apparatus 20 according to the firstembodiment. The camera path displaying system is configured bycomprising an operation unit 1, the information processing apparatus 20,and a displaying unit 11.

The information processing apparatus 20 is, for example, a personalcomputer (main body of PC (personal computer)). The operation unit 1corresponds to a keyboard, a mouse or a dedicated controller of thepersonal computer or the like, acquires an operation input from a user,and transfers the acquired input to the information processing apparatus20. The displaying unit 11 is, for example, a displaying device such asa liquid crystal display, receives display data from the informationprocessing apparatus 20, and displays an image or the like on a screen.

The information processing apparatus 20 according to the presentembodiment has a constitution made by components from athree-dimensional object storing unit 2 to a camera path imagegeneration parameter setting unit 10.

The three-dimensional object storing unit 2 stores data (modeling data,texture data, layout data) related to three-dimensional objects arrangedon a three-dimensional plane in association with the passage of time.

A free viewpoint video image generating unit 3 generates a freeviewpoint video image based on data related to the three-dimensionalobject stored in the three-dimensional object storing unit 2 and camerapath data corresponding to the trajectories of the position and visualfield range (angle of view) of a virtual camera, and outputs thegenerated video image as two-dimensional moving image data.

A camera path editing unit 4 edits the camera path data based on theoperation input from the user via the operation unit 1. Morespecifically, the camera path editing unit 4 edits the camera path databy performing setting and adjustment for the position and visual fieldof the virtual camera based on the operation input from the operationunit 1, with respect to each of frames from the start time to the endtime of a moving image scene. Besides, the camera path editing unit 4has a function of, in a case where the positions and visual fields ofthe virtual cameras are set and adjusted with respect to severalspecific frames based on the operation input of the operation unit 1,obtaining the positions and visual fields of virtual cameras in frameson the way in these specific frames by interpolation.

A camera path storing unit 5 stores the camera path data edited by thecamera path editing unit 4, in association with the passage of time ofthree-dimensional object data.

A camera path image generating unit 8 generates a camera path imagebased on the above-described camera path data and three-dimensionalobject data and a later-described camera path image generationparameter. Details of the camera path image will be described later.

A camera path image storing unit 7 stores the camera path imagegenerated by the camera path image generating unit 8. The camera pathimage is stored in the camera path image storing unit 7 in associationwith a camera path and the camera path image generation parameter usedfor generating the camera path image.

The camera path image generation parameter setting unit 10 sets thecamera path image generation parameter based on the operation input fromthe user via the operation unit 1. The camera path image generationparameter includes parameters corresponding to a position and visualfield looking down on the camera path, and display/non-display, color,shape, display interval and the like of an object to be drawn in thecamera path image.

A camera path image generation parameter storing unit 9 stores thecamera path image generation parameter set by the camera path imagegeneration parameter setting unit 10.

Here, setting or changing of the parameters corresponding to theposition and visual field looking down on the virtual camera isperformed, for example, by operating with the user the position andvisual field of a virtual looking-down camera looking down on thevirtual camera from a predetermined point in the three-dimensional spacevia a controller or the like of the operation unit 1. In this case, whenthe operation of setting or changing the position and visual field ofthe virtual looking-down camera is performed from the user via theoperation unit 1, the camera path image generation parameter settingunit 10 sets or changes the parameters of the position and visual fieldlooking down on the virtual camera, based on the position and visualfield of the virtual looking-down camera.

Besides, setting or changing of the parameters corresponding to thedisplay/non-display, color, shape, display interval and the like of theobject is performed by displaying a parameter operation screen for theseparameters on the screen of the displaying unit 11 and causing the userto perform the operation input via the parameter operation screen. Thatis, when the operation input is performed from the user via theoperation unit 1 to the parameter operation screen, the camera pathimage generation parameter setting unit 10 sets or changes theparameters corresponding to the display/non-display, color, shape,display interval and the like of each drawing object, based on theoperation input.

A camera path list generating unit 6 list-displays the above-describedcamera path images within a window displayed on the screen of thedisplaying unit 11. The camera path list generating unit 6 also has afunction of, together with each list-displayed camera path image,displaying file name, update date and time, metadata (keyword,evaluation score, etc.) and the like of the camera path of each thelist-displayed camera path image, on the displaying unit 11.

Besides, when a selection operation of the camera path image and aselection operation of a processing menu (not illustrated) are performedvia the operation unit 1, the camera path list generating unit 6 starts(activates) to perform the process related to the camera pathcorresponding to the selected camera path image. For example, in a casewhere the camera path images are being list-displayed on the displayingunit 11, when the user selects and operates one camera path image viathe operation unit 1, the camera path list generating unit 6 displaysmenu items such as “edit camera path”, “display free viewpoint videoimage” and the like on the displaying unit 11. Here, when the userselects one of the menu items via the operation unit 1, the camera pathlist generating unit 6 displays a screen corresponding to the selectedmenu item on the displaying unit 11. For example, when the menu item of“edit camera path” is selected, a camera path edit screen for the camerapath is displayed on the displaying unit 11. Further, for example, whenthe menu item of “display free viewpoint video image” is selected, afree viewpoint video image display screen for the camera path isdisplayed on the displaying unit 11.

Besides, the camera path list generating unit 6 also has a function ofdisplaying one or more folders including a camera path file on thedisplaying unit 11. Here, when one folder is selected and operated bythe user via the operation unit 1, the camera path list generating unit6 displays, on the displaying unit 11, a camera path image list in theselected folder.

Besides, the camera path list generating unit 6 also has a function of,in a case where a predetermined search condition of the camera path isinput by a user's menu (not illustrated) operation via the operationunit 1, displaying a camera path image list of the camera path searchedbased on the predetermined search condition. For example, the searchcondition of the camera path includes file name, update date and time,metadata and the like of the camera path.

Besides, the camera path list generating unit 6 also has a function of,in a case where selection operations of the plurality of camera pathimages are performed by the user via the operation unit 1, displaying alist of only the selected camera path images. Thus, in a case where theuser tries to find a desired camera path from among a large number ofcamera paths, it is possible to narrow down and display only similarcamera paths, so that it becomes easy for the user to compare adifference between the respective camera paths narrowed down.

Besides, the camera path list generating unit 6 also has a function ofadding common metadata to the list-displayed camera paths or the camerapaths selected from among the list-displayed camera paths and storingthese camera paths, by a menu (not illustrated) operation and/or acharacter input operation from the user. By adding the metadata to thecamera paths in this manner, it is possible to narrow down the camerapaths by classification or retrieval based on the metadata, so that itbecomes possible for the user to find a desired camera path moreefficiently.

The displaying unit 11 in the camera path displaying system of thepresent embodiment displays the free viewpoint video image, the camerapath list display window, the position and visual field of the virtuallooking-down camera, the operation screen such as the parameteroperation screen or the like, the menu and the like as described above,on the screen.

FIG. 2 is a flowchart for describing a process of generating anddisplaying the above list of the camera path images in the informationprocessing apparatus 20 of the first embodiment. In the followingdescription, each of processing steps S101 to S111 in the flowchart ofFIG. 2 will be abbreviated as S101 to S111 respectively. The process ofthe flowchart of FIG. 2 may be performed by a software configuration ora hardware configuration, or a part of the process may be performed by asoftware configuration and the rest may be realized by a hardwareconfiguration. In the case where the process is performed by thesoftware configuration, for example, a CPU (central processing unit) orthe like executes the program related to the present embodiment storedin a ROM (read only memory) or the like, so that the process isrealized. The program related to the present embodiment may be preparedin advance in the ROM or the like, may be read from a detachablesemiconductor memory or the like, or may be downloaded from a networksuch as the Internet (not illustrated). These are assumed to be the samein other flowcharts to be described later.

In S101, the camera path list generating unit 6 determines whether ornot the list of the camera paths to be list-displayed is updated. Whenit is determined that the list is updated, the camera path listgenerating unit 6 advances the process to S103. On the other hand, whenit is determined that the list is not updated, the camera path listgenerating unit advances the process to S102. Here, for example, when afolder is opened, the camera path list generating unit 6 sets camerapath files in the folder as targets of list display. Besides, in case ofsearching for the camera path by a predetermined search condition (filename, update date and time, metadata, etc.), the camera path listgenerating unit 6 sets the searched camera path files as targets of listdisplay.

In S102, the camera path list generating unit 6 determines whether ornot the camera path image generation parameter is updated. When it isdetermined that the camera path image generation parameter is updated,the camera path list generating unit 6 advances the process to S103. Onthe other hand, when it is determined that the camera path imagegeneration parameter is not updated, the camera path list generatingunit returns the process to S101.

In S103, the camera path list generating unit 6 causes the camera pathimage generating unit 8 to generate a foreground/background image whichis the source of the camera path image. The camera path image generatingunit 8 in S103 generates, as the foreground/background image, an imagewhich is obtained by looking down on the foreground and background ofthe three-dimensional object from a specific viewpoint. The camera pathimage generating unit 8 obtains information on the viewpoint positionand the visual field used for generating the foreground/background imagefrom the camera path image generation parameter. In the presentembodiment, it is assumed that the camera path images to belist-displayed are all images obtained by looking down from the sameviewpoint position and visual field. Besides, in the present embodiment,the foreground/background image generated in S103 is commonly used forall the list-displayed camera path images. When the viewpoint positionor visual field from which the camera path is looked down is changed, itis determined in S102 that the camera path image generation parameter isupdated, so that the foreground/background image is again generated inS103.

Next, in S104, the camera path list generating unit 6 sets a firstcamera path among the camera paths of the camera path images to belist-displayed, as a focused camera path.

Next, in S105, with respect to the target camera path, the camera pathlist generating unit 6 determines whether or not the camera path imagegenerated by the current camera path image generation parameter isstored in the camera path image storing unit 7. When it is determinedthat the camera path image is not stored, the camera path listgenerating unit 6 advances the process to S106. On the other hand, whenit is determined that the camera path image is stored, the camera pathlist generating unit advances the process to S108.

In S106, the camera path list generating unit 6 causes the camera pathimage generating unit 8 to generate the camera path image. The camerapath image generating unit 8 in S106 reads the camera path data from thecamera path storing unit 5, superposes the trajectory of the camera pathon the foreground/background image generated in S103, and generates theobtained image as the camera path image. Here, the information of theposition and visual field of the viewpoint looking down on the camerapath, display/non-display of a line to be superposed, color and line,display interval and the like are obtained from the camera path imagegeneration parameter read from the camera path image generationparameter storing unit 9 or are generated.

Incidentally, since the camera path image data once generated is storedin association with the camera path and the camera path image generationparameter, the camera path image generating process is skipped at thetime of next display, and the stored camera path image data is reused.

In step S107, the camera path image generating unit 8 stores the camerapath image generated in step S106 in the camera path image storing unit7.

In step S108, the camera path list generating unit 6 reads the camerapath image stored in the camera path image storing unit 7.

Then, in S109, the camera path list generating unit 6 lays out thecamera path image read from the camera path image storing unit 7 withinthe window on the screen of the displaying unit 11, and list-displaysthe laid-out camera path images.

Thereafter, in S110, the camera path list generating unit 6 determineswhether or not the focused camera path is the last camera path among thecamera paths being the targets of list display. When it is determined inS110 that the focused camera path is the last camera path, the camerapath list generating unit 6 returns the process to S101 and determineswhether or not the camera path list is updated. On the other hand, whenit is determined in S110 that the focused camera path is not the lastcamera path, the camera path list generating unit 6 advances the processto S111.

In S111, the camera path list generating unit 6 sets, among the camerapaths selected as the targets of list display in S101, a next camerapath of the current focused camera path to be a focused camera path, andreturns the process to S105 to repeat the processes from S105.

By performing the process of FIG. 2 described above, in the camera pathdisplaying system of the present embodiment, when the list of the camerapaths being the targets of list display is updated or when the camerapath image generation parameter is updated, a new camera path image listis displayed.

FIGS. 3A and 3B are diagrams for describing data formats of the camerapaths.

FIG. 3A is a data example in case of designating the three-dimensionalcoordinates of the virtual camera and the three-dimensional coordinatesand angle of view of a virtual camera gaze point for each frame of themoving image scene when generating the camera path. Here, the virtualcamera gaze point is a point which is used to determine a direction thatthe virtual camera faces. The three-dimensional coordinates of thevirtual camera and the three-dimensional coordinates of the virtualcamera gaze point can be controlled independently of each other. Forexample, even if either one or both of the position of the virtualcamera and the virtual camera gaze point move, it is controlled suchthat the virtual camera always faces the virtual camera gaze point.Besides, as the angle of view of the virtual camera, a horizontal angleof view and a vertical angle of view are designated with angles (degreesor radians) respectively. Alternatively, an aspect ratio (horizontal andvertical ratio) of a virtual camera video image and any value of thehorizontal angle of view, the vertical angle of view and a diagonalangle of view are designated. Alternatively, the angle of view may bedesignated as a focal length (mm) in terms of 35 mm film as in case ofrepresenting an angle of view of a general camera lens.

FIG. 3B is another data example in case of designating thethree-dimensional coordinates of the virtual camera, a three-dimensionaldirection in which the virtual camera faces and the angle of view foreach frame of the moving image scene when creating the camera path. In acase where a controller of the operation unit 1 operating the virtualcamera does not have a function of manipulating the position of thevirtual camera gaze point, the data format of FIG. 3B is used. As forthe length of a vector representing the three-dimensional direction,some information may be defined, but if nothing is defined, the relevantlength is assumed to be a unit vector (vector of length 1).

FIGS. 4A and 4B are diagrams for describing a first screen example ofcamera path list display generated by the camera path list generatingunit 6 and displayed on the displaying unit 11. As exemplarilyillustrated in FIG. 4A, the camera path list generating unit 6 generatesthe camera path list display (virtual camera path images) representing aplurality of movement paths including a first movement path (301 in FIG.4A) and a second movement path (302 in FIG. 4A) of the virtualviewpoint. Incidentally, the camera path list generating unit 6generates the camera path list display including a plurality of camerapath images generated by using the camera path data (virtual camera pathinformation).

In FIG. 4A, four camera path images 201, 202, 203 and 204 arelist-displayed in a window 101. The display size of each camera pathimage is set by the camera path image generation parameter. Besides, inthe vicinity of the camera path images 201, 202, 203 and 204, file names211, 212, 213 and 214 of the camera paths of these camera path imagesare associated and displayed respectively.

In each of the camera path images 201, 202, 203 and 204, a foreground401 and a background 411 respectively based on the three-dimensionalobject data are displayed. The three-dimensional objects of theforeground 401 and the background 411 are generated by, for example,separating via an image process the foreground (players, ball, etc.) andthe background (field, spectators' seats, etc.) from the video imagesobtained by shooting a sports game with a large number of cameras(actual cameras instead of virtual cameras). Incidentally, all or a partof the three-dimensional objects may be generated by computer graphics.Display/non-display of the foreground and background is set by thecamera path image generation parameter. The foreground and backgroundmay change depending on the progress of a scene, but in the presentembodiment the foreground and background of a specific frame are used asrepresentatives.

The specific frame is set based on the camera path image generationparameter. As the specific frame, for example, a start frame or an endframe of the moving image scene may be used, or an arbitrary frame inthe middle may be used. The camera path image includes at least one of astate of the virtual camera in the specific frame, and the foregroundand background of the three-dimensional space.

Besides, a plurality of frames may be designated as the specific frames.In a case where the plurality of frames are designated as the specificframes, the camera path image generating unit 8 may generate the camerapath image by multiplexing the foregrounds of the designated pluralityof frames. At that time, for example, it may be possible to cause theuser to easily understand the passage of time by gradually changingopacity of the foreground (for example, increasing the opacity) from thestart to the end of the moving image scene. Alternatively, it may bepossible to rank the plurality of frames according to degrees ofimportance and increase opacity of the foreground of the high-rankingframe to achieve clear display. For example, in a soccer goal scene,when the foreground image at the moment of making a goal is displayedmost clearly, it may be possible to cause the user to easily understandthat the viewpoint at the relevant moment is particularly important.

In the camera path image generating unit 8, the foreground 401 and thebackground 411 are rendered as two-dimensional images which are obtainedby looking down on the three-dimensional space from a specificviewpoint. The looking-down viewpoint position and the range of thevisual field are set by the camera path image generation parameters. Asthe looking-down viewpoint, for example, it may be possible to set aviewpoint for looking down on the center point of a stadium fromdirectly above may be set. Alternatively, as illustrated in FIG. 4A, itmay be possible to set a viewpoint for looking down an important pointin the relevant scene diagonally from above.

FIG. 5 is a diagram for describing a screen example of the four camerapath images 201 to 204 which are list-displayed by the camera path listgenerating unit 6 after an operation of simultaneously changing thelooking-down viewpoint positions for the four camera path images 201 to204 of FIG. 4A was performed and thus these images were updated.Display/non-display of the foreground 401 and the background 411 in FIG.5 is set by the camera path image generation parameter.

As illustrated in FIGS. 4A and 5, figures (hereinafter referred to astrajectories 301, 302, 303 and 304) representing movement paths of thevirtual camera are displayed respectively with respect to the camerapath images 201, 202, 203 and 204 respectively. In the trajectorydisplay, it is possible to plot marks at specific frame intervals andrepresent movement speed of the virtual cameras by sparseness anddensity of the plotted marks. Besides, in the case where the foregroundsof the specific plurality of frames are multiplexed and displayed asdescribed above, it may be possible to plot marks on the positions ofthe virtual cameras in the plurality of frames. Besides, in the casewhere the opacity of the foregrounds in the above specific plurality offrames is changed and displayed, it may be possible to display the marksto be plotted by changing their opacity in conformity with the opacityof the foregrounds.

Display/non-display of a line, color and shape of the line, displayinterval of marks, and the like as figures representing the trajectoryare set by the camera path image generation parameters. Incidentally, inthe trajectory display, since the three-dimensional trajectory isprojected on a two-dimensional plane, it may be difficult to understanda height or a perspective. In the case where it is difficult tounderstand the height or the perspective, an auxiliary line may bedisplayed as a figure representing the trajectory as appropriate. As theauxiliary line, for example, a perpendicular line may be drawn from apoint on the trajectory toward the ground (virtual ground).Alternatively, the trajectory of the intersection between theperpendicular line and the ground may be displayed as the auxiliaryline. Drawing the auxiliary line like this makes it easier to understandthe height and the perspective. Display/non-display of or not theauxiliary line, color and shape of the auxiliary line, and displayinterval of the auxiliary line are set by the camera path imagegeneration parameter.

Besides, as illustrated in FIGS. 4A and 5, a scroll bar 501 to beslide-operated in response to an operation of the operation unit 1 isdisplayed on the screen. When the scroll bar 501 is slide-operated, forexample, in the downward direction by the operation of the operationunit 1, the camera path list generating unit 6 causes the window 101 tobe scroll-displayed, so that further four camera path images 205, 206,207 and 208 are list-displayed as illustrated in FIG. 4B.

Also in the example of FIG. 4B, as well as FIG. 4A, the display size ofthe four camera path images 205 to 208 is set by the camera path imagegeneration parameter. Besides, in the vicinity of the camera path images205, 206, 207 and 208, file names 215, 216, 217 and 218 of the camerapaths of these camera path images are associated and displayedrespectively. Similarly to the above, the foreground and backgroundbased on the three-dimensional object data and figures (trajectories305, 306, 307 and 308) representing movement paths of the virtual cameraare displayed respectively in the camera path images 205 to 208.Similarly to the above, also in the example of FIG. 4B, the scroll bar501 to be slide-operated in response to the operation of the operationunit 1 is displayed on the screen, and scroll display of the window 101is performed according to the slide operation.

FIG. 6 is a diagram for describing a second screen example of the camerapath list display generated by the camera path list generating unit 6and displayed on the displaying unit 11.

In the second screen example, the camera path list generating unit 6causes the camera path image generating unit 8 to generate the camerapath images which are obtained by adding figures (trajectories 321, 322,323 and 324) representing the movement paths of the virtual camera gazepoints to the respective display contents of the first screen example ofFIG. 4A. The display of the screen example of FIG. 6 can be performed ina case where the data format of the camera path includes thethree-dimensional coordinates of the virtual camera gaze point (see FIG.3A). Also in case of displaying the trajectories 321 to 324 of thevirtual camera gaze point positions, as well as the above example of themovement path of the virtual camera, it is possible to plot marks atspecific frame intervals and represent movement speed of the virtualcameras by sparseness and density of the plotted marks.Display/non-display of a line being the figure representing the movementpath of the virtual camera gaze point, color and shape of the line,display interval of the marks, and the like are set by the camera pathimage generation parameters.

FIGS. 7A and 7B are diagrams for describing a third screen example ofthe camera path list display generated by the camera path listgenerating unit 6 and displayed on the displaying unit 11. FIG. 7B isthe diagram obtained by enlarging a part (e.g., the camera path image201) of FIG. 7A.

In the third screen example, the camera path list generating unit 6causes the camera path image generating unit 8 to generate the camerapath images which are obtained by adding figures (arrow lines 331, 332,333, 334, 335 and 336) representing three-dimensional directions towhich the virtual cameras face to the respective display contents of thefirst screen example of FIG. 4A. The display of the arrow lines 331 to336 representing the three-dimensional directions to which the virtualcameras face can be performed also in a case where the data format ofthe camera path does not include the three-dimensional coordinates ofthe virtual camera gaze point (see FIG. 3B). Display/non-display of thearrow line being the figure representing the three-dimensional directionto which the virtual camera faces, color and shape of the arrow line,display interval of the arrow lines are set by the camera path imagegeneration parameter.

FIG. 8 is a diagram for describing a fourth screen example of the camerapath list display generated by the camera path list generating unit 6and displayed on the displaying unit 11. FIG. 8 is the diagram obtainedby enlarging a part (e.g., the camera path image 201) of the camera pathlist display similar to that in FIG. 7A.

The fourth screen example of FIG. 8 is a display example in which thethicknesses of arrow lines 341, 342, 343, 344, 345 and 346 representingthe three-dimensional directions to which the virtual cameras face arechanged with respect to the screen example of FIG. 7B. In the fourthscreen example, the line thickness of each of the arrow lines 341 to 346represents the angle of view of the virtual camera, and the thicker thearrow line, the wider the angle of view. Incidentally, as a modifiedexample of FIG. 8, the angle of view of the virtual camera may berepresented by the color density of the arrow line, the size of the tipof the arrow, and the like. Association between the thickness of thearrow line (or the color density of the arrow line, the size of the tipof the arrow) and the angle of view of the virtual camera is set by thecamera path image generation parameter.

FIG. 9 is a diagram for describing a further modified example of FIG. 8,and is a screen example in which each of angles of view 351, 352, 353,354, 355 and 356 of the virtual camera is represented by an openingangle of two lines. Also in the screen example of FIG. 9, associationbetween the opening angle of the two lines and the angle of view of thevirtual camera is set by the camera path image generation parameter.

FIG. 10 is a diagram for describing a further modified example of FIG.8, and is a screen example in which horizontal angles of view andvertical angles of view are represented by quadrangular pyramids 361,362 and 363. The center of the bottom side of each of the quadrangularpyramids 361, 362 and 363 represents the virtual camera gaze point.Display of the screen example of FIG. 10 can be performed in the casewhere the data format of the camera path includes the three-dimensionalcoordinates of the virtual camera gaze point (see FIG. 3A).

Incidentally, in the screen examples of FIGS. 9 and 10, the angle ofview can be more intuitively represented as compared with the screenexample of FIG. 8. However, since the number of lines for representingthe angle of view increases, there is a possibility that the displaybecomes complicated.

Since switching of displaying method of the angle of view is set by thecamera path image generation parameter, the user can appropriatelyswitch to an eye-friendly displaying method by the operation via theoperation unit 1. In this case, the camera path list generating unit 6switches the display in accordance with the operation via the operationunit 1.

In the above screen example, the foreground/background image isgenerated from the three-dimensional object, and the figure representingthe trajectory of the camera path is superposed on theforeground/background image. As another example, it may be possible togenerate the camera path image by superposing a figure representing thetrajectory of the camera path on a video image of an actual camera(preferably looking down on the entire range where a subject ispresent). In this case, there are restrictions on the viewpoint positionand visual field looking down on the camera path. However, thethree-dimensional object storing unit 2 becomes unnecessary and theimage process of generating the foreground/background image from thethree-dimensional object becomes unnecessary in the camera path imagegenerating unit 8, so that the system can be simplified. When the camerapath image is selected and the free viewpoint video image is displayedon the camera path list display screen, it only has to previously storethe free viewpoint video image generated by the camera path as themoving image file and play back the stored video image.

When there are the plurality of above actual cameras and the respectivelooking-down viewpoint positions and visual fields are different inthese cameras, it may be possible to generate the camera path image byusing the camera designated by the camera path image generationparameter. Besides, for example, in the generation of the camera pathimage of sports broadcasting, it may be possible to display thepositions and motions of players, balls and the like with graphics(lines, figures, numbers, etc.) by obtaining tracking data.

As described above, according to the first embodiment, bylist-displaying the images (camera path images) obtained by looking downon the trajectories of the camera paths, it is possible for the user toquickly find the desired camera path from among the large number ofcamera paths. In particular, in a case where there are a large number ofsimilar camera paths, by firstly looking at the camera path images tonarrow down the candidates and further changing the viewpoint positionand visual field for looking down on the camera path, the user caneasily distinguish the similar camera paths. Furthermore, if it is stilldifficult to distinguish the similar camera paths, the user may selectthe candidate camera path image and actually play back the freeviewpoint video image. As just described, according to the firstembodiment, the user can efficiently find the desired camera path bylooking at the camera path images to narrow down the candidates and thenplaying back the free viewpoint video images, rather than by playingback the free viewpoint video image for the large number of camera pathseach time.

Besides, in the present embodiment, when the operation to change thecamera path image generation parameter is performed, the change isapplied all at once to all the camera path images being the listtargets. Therefore, the user can easily perform the work of finding thedesired camera path by comparing the plurality of camera paths under thesame condition.

Second Embodiment

Hereinafter, an example in which the camera path image generationparameter can be set for each camera path will be described as thesecond embodiment. Since the configuration of the camera path displayingsystem of the second embodiment is the same as that of FIG. 1,illustration and explanation thereof will be omitted. In a case whereone or more camera path images are selected and the camera path imagegeneration parameter is set by a menu (not illustrated) operation, thecamera path list generating unit 6 of the second embodiment changes onlythe camera path image generation parameter corresponding to the selectedcamera path.

FIG. 11 is a flowchart for describing a flow of the process oflist-displaying the camera path images in the second embodiment.

In FIG. 11, processes in S201 and S202 are respectively the same asthose in S101 and S102 of FIG. 2, a process in S203 is the same as thatin S104, a process in S204 is the same as that in S105, and processes inS206 to S211 are respectively the same as those in S106 to S111.

In the flowchart of FIG. 11, when it is determined in S201 that thecamera path list is updated, or when it is determined in S202 that thecamera path image generation parameter is updated, the process isadvanced to S203. Besides, when it is determined in S204 that the camerapath image is not stored, the process is advanced to S205.

In S205, the foreground/background image generating process is performedfor each camera path by the camera path image generating unit 8, andthereafter the process is advanced to S206. When superposing the camerapath trajectory in S206, display/non-display of the trajectory, colorand shape of the line, display interval and the like are determinedbased on the camera path image generation parameter for each camerapath.

That is, in the flowchart of FIG. 2 of the first embodiment, the camerapath list generating unit 6 causes the camera path image generating unit8 to perform the foreground/background image generating process of S103in common for all the camera paths. On the other hand, in the flowchartof FIG. 11 of the second embodiment, the camera path list generatingunit 6 causes the camera path image generating unit 8 to perform theforeground/background image generating process for each camera path inS205. Besides, in S206, the camera path list generating unit 6 causesthe camera path image generating unit 8 to superpose a figurerepresenting the trajectory on the foreground/background image based onthe camera path image generation parameter for each camera path.

In the above flowchart of FIG. 2, it is aimed to avoid duplication ofthe same process by previously creating the common camera path image inS103. Also in the flowchart of FIG. 11, when generating theforeground/background image in S205, it is determined whether or not thecamera path images based on the same camera path image generationparameter are stored. Then, when the camera path images based on thesame camera path image generation parameter are stored, it is possibleto avoid duplication of the same process.

FIG. 12 is a diagram for describing an updated screen example which isobtained after an operation of changing the looking-down viewpointpositions was performed for the camera path image 201 and the camerapath image 204 of the screen example illustrated in FIG. 4A. A camerapath image 201 a of FIG. 12 is an image obtained after the operation ofchanging the looking-down viewpoint position for the camera path image201 of FIG. 4A was performed. A camera path image 204 a of FIG. 12 is animage obtained after the operation of changing the looking-downviewpoint position for the camera path image 204 of FIG. 4A wasperformed. When the looking-down viewpoint positions are changed, asshown by the camera path images 201 a and 204 a of FIG. 12, not only theforeground 401 and the background 411 but also the trajectories 301 and304 of the virtual camera positions are changed according to the updatedlooking-down viewpoint positions.

Besides, in the second embodiment, in the case where the camera pathimage which well represents the feature of the camera path is generatedbased on the setting of the image generation parameter, it may bepossible to store the relevant camera path image as the default camerapath image of the camera path. Thus, when the camera path list isdisplayed for the first time, it is possible to display the camera pathimage which well represents the feature of the camera path, so that theuser can easily narrow down the camera paths when looking at the camerapath image list.

For example, in the same scene of sports, a camera path group which paysattention to a player A uses as a default the camera path image obtainedby looking down around the player A, and a camera path group which paysattention to a player B uses as a default the camera path image obtainedby looking down around the player B. Thus, when the user looks at thecamera path image list of the relevant scene, he/she can easily narrowdown to the camera path focusing on, e.g., the player A.

Besides, for example, in the same scene of sports, a defaultlooking-down direction may be changed between a camera path group payingattention to an attacking-side team and a camera path group payingattention to a defending-side team. Thus, when the user looks at thecamera path image list of the relevant scene, he/she can easily narrowdown to the camera path which pays attention to one of these teams.

As described above, in the second embodiment, by enabling to set thecamera path image generation parameter for each camera path, the usercan easily grasp the feature for each camera path when he/she looks atthe camera path image list display.

As described above, according to the first and second embodiments, thedisplay and updating of the image obtained by looking down on thetrajectory of the camera path are performed, so that the user canquickly confirm the content of each camera path. Therefore, the user canquickly find the desired camera path from among the large number ofcamera paths. In the above embodiment, as illustrated in FIGS. 4A to 7B,the example that the plurality of camera path images are displayed sideby side has been mainly described, but the present invention is notlimited to this example. For example, it is also possible to adopt amode in which two or more camera path images are superposed anddisplayed in one video image.

Although the present invention has been described as above inconjunction with the above embodiments, these embodiments are merely theexamples of concretization for carrying out the present invention.Accordingly, the technical scope of the present invention should not beinterpreted restrictively or limitedly by the above embodiments. Namely,the present invention can be carried out in various forms withoutdeparting from the technical idea or the main feature of the presentinvention.

According to the above information processing apparatus 20, it ispossible to quickly find the desired camera path from among theplurality of camera paths.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2017-223686, filed Nov. 21, 2017, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. A display controlling apparatus comprising: oneor more hardware processors; and one or more memories which storeinstructions executable by the one or more hardware processors to causethe display controlling apparatus to perform at least: (1) obtainingviewpoint path information for specifying a movement path of a virtualviewpoint corresponding to a virtual viewpoint image that is generatedbased on a plurality of captured images of an area captured with aplurality of image capturing apparatuses; (2) generating, based on theobtained viewpoint path information, a path representation image forrepresenting concurrently a plurality of movement paths of the virtualviewpoint; and (3) causing a display screen to display the generatedpath representation image, wherein in the generated path representationimage, a figure representing a movement path of the virtual viewpoint issuperimposed on an image of the area viewed from a specific viewpointthat is not on a movement path of the virtual viewpoint.
 2. The displaycontrolling apparatus according to claim 1, wherein the instructionsfurther cause the display controlling apparatus to perform: setting aparameter for generating the path representation image based on userinput, wherein the path representation image is generated based on theobtained viewpoint path information and the set parameter.
 3. Thedisplay controlling apparatus according to claim 1, wherein the pathrepresentation image includes a figure representing a time-dependentchange of at least any of a position of the virtual viewpoint, adirection of the virtual viewpoint, an angle of view of the virtualviewpoint, and a gaze point corresponding to the virtual viewpoint. 4.The display controlling apparatus according to claim 1, wherein theinstructions further cause the display controlling apparatus to performobtaining data related to a three-dimensional object in the area, andwherein in the generating, the path representation image including animage of the three-dimensional object in the area is generated based onthe obtained data.
 5. The display controlling apparatus according toclaim 1, wherein the path representation image includes a figurerepresenting at least any of height and perspective of the movement pathof the virtual viewpoint.
 6. The display controlling apparatus accordingto claim 2, wherein the set parameter for generating the pathrepresentation image includes a parameter representing a position of aviewpoint for looking on a movement path of the virtual viewpoint. 7.The display controlling apparatus according to claim 2, wherein the setparameter includes information for designating at least any ofdisplay/non-display, shape, color, display interval, and opacity of afigure or an image displayed in the path representation image.
 8. Thedisplay controlling apparatus according to claim 2, wherein theparameter includes information for designating a point of time in acapturing period of the plurality of image capturing apparatuses.
 9. Thedisplay controlling apparatus according to claim 8, wherein thegenerated path representation image represents both an object in thearea at the designated point of time and the plurality of movement pathsof the virtual viewpoint in a period including time around thedesignated point of time.
 10. The display controlling apparatusaccording to claim 9, wherein the path representation image is generatedby superimposing images of objects at a plurality of designated pointsof time.
 11. The display controlling apparatus according to claim 10,wherein in the generated path representation image, the opacity of theobjects is gradually changed depending on the point of timecorresponding to each object.
 12. The display controlling apparatusaccording to claim 10, wherein in the generated path representationimage, the opacity of an object is determined based on a degree ofimportance of the point of time corresponding to the object.
 13. Thedisplay controlling apparatus according to claim 1, wherein, in a casewhere at least one movement path among the plurality of movement pathsis updated, the path representation image is updated.
 14. The displaycontrolling apparatus according to claim 2, wherein the pathrepresentation image includes a plurality of path images respectivelyrepresenting the plurality of movement paths of the virtual viewpoint,and wherein in a case where the parameter is updated by user input, theupdated parameter is applied to all of the plurality of path images. 15.The display controlling apparatus according to claim 2, wherein the pathrepresentation image includes a plurality of path images respectivelyrepresenting the plurality of movement paths of the virtual viewpoint,and wherein in a case where the parameter is updated by user input, theupdated parameter is applied to each of one or more path images selectedfrom among the plurality of path images.
 16. A display controllingmethod comprising: obtaining viewpoint path information for specifying amovement path of a virtual viewpoint corresponding to a virtualviewpoint image that is generated based on a plurality of capturedimages of an area captured with a plurality of image capturingapparatuses; generating, based on the obtained viewpoint pathinformation, a path representation image for representing concurrently aplurality of movement paths of the virtual viewpoint; and causing adisplay screen to display the generated path representation image,wherein in the generated path representation image, a figurerepresenting a movement path of the virtual viewpoint is superimposed onan image of the area viewed from a specific viewpoint that is not on amovement path of the virtual viewpoint.
 17. The display controllingmethod according to claim 16, further comprising setting a parameter forgenerating the path representation image based on user input, whereinthe path representation image is generated based on the obtainedviewpoint path information and the set parameter.
 18. A non-transitorycomputer-readable storage medium which stores a program for causing acomputer to perform a display controlling method comprising: obtainingviewpoint path information for specifying a movement path of a virtualviewpoint corresponding to a virtual viewpoint image that is generatedbased on a plurality of captured images of an area captured with aplurality of image capturing apparatuses; generating, based on theobtained viewpoint path information, a path representation image forrepresenting concurrently a plurality of movement paths of the virtualviewpoint; and causing a display screen to display the generated pathrepresentation image, wherein in the generated path representationimage, a figure representing a movement path of the virtual viewpoint issuperimposed on an image of the area viewed from a specific viewpointthat is not on a movement path of the virtual viewpoint.
 19. The storagemedium according to claim 18, wherein the display controlling methodfurther comprises setting a parameter for generating the pathrepresentation image based on user input, and wherein the pathrepresentation image is generated based on the obtained viewpoint pathinformation and the set parameter.
 20. The display controlling apparatusaccording to claim 1, wherein the image of the area is obtained by animage capturing apparatus positioned at the specific viewpoint.
 21. Thedisplay controlling apparatus according to claim 1, wherein the image ofthe area is generated based on user input designating the specificviewpoint and the plurality of images of the area captured with theplurality of image capturing apparatuses.
 22. The display controllingapparatus according to claim 1, wherein the instructions further causethe display controlling apparatus to perform: receiving user input forselecting a movement path among the plurality of movement paths of thevirtual viewpoint; and causing the display screen to display a virtualviewpoint image corresponding to the selected movement path of thevirtual viewpoint.
 23. The display controlling apparatus according toclaim 1, wherein the generated path representation image includes aplurality of path images each representing a movement path of thevirtual viewpoint, and wherein in each of the plurality of path images,a figure representing a movement path of the virtual viewpoint issuperimposed on an image of the area viewed from a specific viewpointthat is not on a movement path of the virtual viewpoint.